Robotic surgical systems with roll, pitch, and yaw realignment including trim and flip algorithms

ABSTRACT

A robotic surgical system or simulator includes an input device, a display device, and a processing unit. The display device includes a representation of a surgical tool that is operably associated with the input device. The processing unit is in communication with the input device and is associated with the representation of a surgical tool to rotate the representation about a first axis of movement based on a scaled rotation of the input device about a first axis of rotation. In an aligned configuration, the input device is aligned with the representation about the first axis of rotation. When the input device is misaligned with the representation, the processing unit varies the scaled rotation of the input device to return the input device to the first aligned configuration until the input device is misaligned about the first axis of rotation a first predetermined offset from the aligned configuration.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Stage Application filed under 35U.S.C. § 371(a) of International Patent Application Serial No.PCT/US2018/049330, filed Sep. 4, 2018, which claims the benefit of andpriority to U.S. Provisional Patent Application Ser. No. 62/554,292,filed Sep. 5, 2017, the entire disclosure of which is incorporated byreference herein.

BACKGROUND

Robotic surgical systems have been used in minimally invasive medicalprocedures. During a medical procedure, the robotic surgical system iscontrolled by a surgeon interfacing with a user interface. The userinterface allows the surgeon to manipulate an end effector that acts ona patient. The user interface includes an input controller or handlethat is moveable by the surgeon to control the robotic surgical system.

Robotic surgical systems typically used a scaling factor to scale downthe motions of the surgeons hands to determine the desired position ofthe end effector within the patient so that the surgeon could moreprecisely move the end effector inside the patient. However, the largerthe scaling factor, the farther the surgeon had to move the input devicehandle to move the end effector the same distance. Since the inputdevice handle has a fixed range of motion, this meant that for largerscaling factors the surgeon may have reached an end of the range ofmotion of an input handle more often.

In addition, during a medical procedure a surgeon needs to rotate theend effector about a roll axis, a pitch axis, and a yaw axis to properlyposition the end effector to act on tissue. Further, during a medicalprocedure, clutching of movement of the input handle relative to theinput handle may cause the input handle to become misaligned with theend effector about one or more of the roll axis, the pitch axis, and/orthe yaw axis.

There is a continuing need for a robotic surgical system that realignsthe input handle with the end effector during a medical procedure.

SUMMARY

This disclosure generally relates to the scaling of movement of an inputdevice of a user interface to movement of a tool of a robotic systemduring a surgical procedure about one or more of roll, pitch, and yawaxis including a “trim” and/or a “flip” algorithm.

In an aspect of the present disclosure, a robotic surgical system orsimulator includes an input device, a display device, and a processingunit. The input device is rotatable about a first axis of rotation. Thedisplay device includes a representation of a surgical tool that isoperably associated with the input device. The processing unit is incommunication with the input device and is operatively associated withthe representation of a surgical tool to rotate the representation of asurgical tool about a first axis of movement based on a scaled rotationof the input device about the first axis of rotation. The input devicehas an aligned configuration in which the input device is aligned withthe representation of a surgical tool about the first axis of rotation.When the input device is misaligned with the representation of asurgical tool about the first axis of rotation, the processing unitvaries the scaled rotation of the input device about the first axis ofrotation to return the input device to the first aligned configurationuntil the input device is misaligned about the first axis of rotation bya first predetermined offset from the aligned configuration. The firstpredetermined offset may be in a range of about 5° to about 45°.

In aspects, the input device is rotatable about a second axis ofrotation that is perpendicular to the first axis of rotation. Theprocessing unit may be in communication with the input device and may beoperably associated with the representation of a surgical tool to rotatethe representation of a surgical tool about a second axis of movementbased on a scaled rotation of the input device about the second axis ofrotation. In the aligned configuration, the input device is aligned withthe representation of a surgical tool about the second axis of rotation.When the input device is misaligned with the representation of asurgical tool about the second axis of rotation, the processing unitvaries the scaled rotation of the input device about the second axis ofrotation to return the input device to the aligned configuration untilthe input device is misaligned about the second axis of rotation asecond predetermined offset from the aligned configuration. The firstpredetermined offset may be equal to the second predetermined offset.Alternatively, the first predetermined offset may be greater or lessthan the second predetermined offset. For example, the firstpredetermined offset may be in a range of about 16° to about 45° and thesecond predetermined offset may be in a range of about 5° to about 15°.

In some aspects, the input device is rotatable about a third axis ofrotation that is perpendicular to the first and second axes of rotation.The processing unit may be in communication with the input device andmay be operably associated with the representation of a surgical tool torotate the representation of a surgical tool about a third axis ofmovement based on a scaled rotation of the input device about the thirdaxis of rotation. In the aligned configuration the input device isaligned with the representation of a surgical tool about the third axisof rotation. When the input device is misaligned with the representationof a surgical tool about the third axis of rotation, the processing unitvaries the scaled rotation of the input device about the third axis ofrotation to return the input device to the aligned configuration untilthe input device is misaligned about the third axis of rotation a thirdpredetermined offset from the aligned configuration.

In particular aspects, the first predetermined offset is equal to eachof the second and third predetermined offsets. Alternatively, the firstpredetermined offset may be greater or less than the second or thirdpredetermined offsets. Additionally, the second predetermined offset maybe greater or less than the second predetermined offset. The first,second, and/or third predetermined offset may be selectable by a user.Additionally or alternatively, the first, second, and/or thirdpredetermined offsets may be at least partially determined based on therepresentation of a tool. When the input device is misaligned with therepresentation of a surgical tool about the first, second, and/or thirdaxis of rotation by an amount greater than a predetermined misalignment,the aligned configuration about a respective one of the first, second,and/or third axis of rotation may be flipped 180° about the respectiveaxis of rotation.

In another aspect of the present disclosure, a method of operating asurgical robot or surgical simulator includes a processing unitreceiving a rotation of an input device of a robotic surgical systemabout a first axis of rotation and scaling the rotation of the inputdevice to a rotation of a representation of a tool on a display deviceabout a first axis of movement. The processing unit scales down rotationof the input device when the input device is moved away from an alignedconfiguration to realign the input device with the representation of atool until the input device is within a predetermined offset with therepresentation of a tool about the first axis of rotation.

In aspects, scaling rotation of the input device to the rotation of therepresentation of a tool on the display device includes the processingunit scales up rotation of the input device when the input device ismoved towards the aligned configuration to realign the input device withthe representation of a tool until the input devices is within thepredetermined offset with the representation of a tool about the firstaxis of rotation. The method may include selecting the predeterminedoffset. Additionally or alternatively, selecting the predeterminedoffset may include the processing unit determining the predeterminedoffset based on the representation of a tool.

In some aspects, the method includes flipping the aligned configurationof the input device 180° about the first axis of rotation when the inputdevice is misaligned with the representation of the tool greater than apredetermined misalignment.

Further details and aspects of exemplary embodiments of the presentdisclosure are described in more detail below with reference to theappended figures.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of the present disclosure are described herein belowwith reference to the drawings, which are incorporated in and constitutea part of this specification, wherein:

FIG. 1 is a schematic illustration of an user interface and a roboticsystem in accordance with the present disclosure;

FIG. 2 is a perspective view of a input device supported on an end of acontrol arm of the user interface of FIG. 1 ;

FIG. 3 is a view of a display device of the robotic system of FIG. 1illustrating a representation of a tool within a surgical site while ina first position; and

FIG. 4 is a view of the display device of FIG. 3 illustrating therepresentation of the tool of FIG. 3 rotated about 180° about a rollaxis thereof.

DETAILED DESCRIPTION

Embodiments of the present disclosure are now described in detail withreference to the drawings in which like reference numerals designateidentical or corresponding elements in each of the several views. Asused herein, the term “clinician” refers to a doctor, a nurse, or anyother care provider and may include support personnel. Throughout thisdescription, the term “proximal” refers to the portion of the device orcomponent thereof that is closest to the clinician and the term “distal”refers to the portion of the device or component thereof that isfarthest from the clinician. In addition, as used herein the term“neutral” is understood to mean non-scaled.

This disclosure generally relates to the scaling of movement of an inputdevice of a user interface for movement of a tool of a robotic systemduring a surgical procedure. In particular, this disclosure relates tothe scaling of movement about a roll axis, a pitch axis, and a yaw axisof the tool. The robotic system includes an alignment algorithmconfigured to realign the input device of a user interface with theposition of the tool about one or more of the roll, pitch, and yaw axes.In addition, the robotic system may include a “trim” algorithm thatallows the input device to remain offset at a predetermined offset aboutone or more of the roll, pitch, and yaw axes instead of fully aligningthe input device about each of the roll, pitch, and yaw axes.Additionally or alternatively, the robotic system may include a “flip”algorithm that rotates an aligned configuration of the input devicerelative to the tool 180° about a respective one of the roll, pitch, andyaw axes when the input device is misaligned greater than apredetermined misalignment about a respective one of the roll, pitch,and yaw axes.

Referring to FIG. 1 , a robotic surgical system 1 in accordance with thepresent disclosure is shown generally as a robotic system 10, aprocessing unit 30, and a user interface 40. The robotic system 10generally includes linkages 12 and a robot base 18. The linkages 12moveably support an end effector or tool 20 which is configured to acton tissue. The linkages 12 may be in the form of arms each having an end14 that supports an end effector or tool 20 which is configured to acton tissue. In addition, the ends 14 of the arms 12 may include animaging device 16 for imaging a surgical site “S”. The user interface 40is in communication with robot base 18 through the processing unit 30.

The user interface 40 includes a display device 44 which is configuredto display three-dimensional images. The display device 44 displaysthree-dimensional images of the surgical site “S” which may include datacaptured by imaging devices 16 positioned on the ends 14 of the arms 12and/or include data captured by imaging devices that are positionedabout the surgical theater (e.g., an imaging device positioned withinthe surgical site “S”, an imaging device positioned adjacent the patient“P”, imaging device 56 positioned at a distal end of an imaging arm 52).The imaging devices (e.g., imaging devices 16, 56) may capture visualimages, infra-red images, ultrasound images, X-ray images, thermalimages, and/or any other known real-time images of the surgical site“S”. The imaging devices transmit captured imaging data to theprocessing unit 30 which creates three-dimensional images of thesurgical site “S” in real-time from the imaging data and transmits thethree-dimensional images to the display device 44 for display.

The user interface 40 also includes input handles 42 which are supportedon control arms 43 which allow a clinician to manipulate the roboticsystem 10 (e.g., move the arms 12, the ends 14 of the arms 12, and/orthe tools 20). Each of the input handles 42 is in communication with theprocessing unit 30 to transmit control signals thereto and to receivefeedback signals therefrom. Additionally or alternatively, each of theinput handles 42 may include input devices 46 (FIG. 2 ) which allow thesurgeon to manipulate (e.g., clamp, grasp, fire, open, close, rotate,thrust, slice, etc.) the tools 20 supported at the ends 14 of the arms12.

With additional reference to FIG. 2 , each of the input handles 42 ismoveable through a predefined workspace to move the ends 14 of the arms12, e.g., tools 20, within a surgical site “S”. The three-dimensionalimages on the display device 44 are orientated such that the movement ofthe input handles 42 move the ends 14 of the arms 12 as viewed on thedisplay device 44. The three-dimensional images remain stationary whilemovement of the input handles 42 is scaled to movement of the ends 14 ofthe arms 12 within the three-dimensional images. To maintain anorientation of the three-dimensional images, kinematic mapping of theinput handles 42 is based on a camera orientation relative to anorientation of the ends 14 of the arms 12. The orientation of thethree-dimensional images on the display device 44 may be mirrored orrotated relative to view from above the patient “P”. In addition, thesize of the three-dimensional images on the display device 44 may bescaled to be larger or smaller than the actual structures of thesurgical site permitting a clinician to have a better view of structureswithin the surgical site “S”. As the input handles 42 are moved, thetools 20 are moved within the surgical site “S” as detailed below.Movement of the tools 20 may also include movement of the ends 14 of thearms 12 which support the tools 20.

For a detailed discussion of the construction and operation of a roboticsurgical system 1, reference may be made to U.S. Pat. No. 8,828,023, theentire contents of which are incorporated herein by reference.

As detailed above, the user interface 40 is in operable communicationwith the robotic system 10 to perform a surgical procedure on a patient;however, it is envisioned that the user interface 40 may be in operablecommunication with a surgical simulator (not shown) to virtually actuatea robotic system and/or tool in a simulated environment. For example,the robotic surgical system 1 may have a first mode in which the userinterface 40 is coupled to actuate the robotic system 10 and a secondmode in which the user interface 40 is coupled to the surgical simulatorto virtually actuate a robotic system. The surgical simulator may be astandalone unit or be integrated into the processing unit 30. Thesurgical simulator virtually responds to a clinician interfacing withthe user interface 40 by providing visual, audible, force, and/or hapticfeedback to a clinician through the user interface 40. For example, as aclinician interfaces with the input handles 42, the surgical simulatormoves representative tools that are virtually acting on tissue. It isenvisioned that the surgical simulator may allow a clinician to practicea surgical procedure before performing the surgical procedure on apatient. In addition, the surgical simulator may be used to train aclinician on a surgical procedure. Further, the surgical simulator maysimulate “complications” during a proposed surgical procedure to permita clinician to plan a surgical procedure.

The movement of the tools 20 is scaled relative to the movement of theinput handles 42. When the input handles 42 are moved within apredefined workspace, the input handles 42 send control signals to theprocessing unit 30. The processing unit 30 analyzes the control signalsto move the tools 20 in response to the control signals. The processingunit 30 transmits scaled control signals to the robot base 18 to movethe tools 20 in response to the movement of the input handles 42. Theprocessing unit 30 scales the control signals by dividing anInput_(distance) (e.g., the distance moved by one of the input handles42) by a scaling factor S_(F) to arrive at a scaled Output_(distance)(e.g., the distance that one of the ends 14 is moved). The scalingfactor S_(F) is in a range between about 1 and about 10 (e.g., 3). Thisscaling is represented by the following equation:Output_(distance)=Input_(distance) /S _(F)It will be appreciated that the larger the scaling factor S_(F) thesmaller the movement of the tools 20 relative to the movement of theinput handles 42.

For a detailed description of scaling movement of the input handle 42along the X, Y, and Z coordinate axes to movement of the tool 20,reference may be made to commonly owned International Patent ApplicationSerial No. PCT/US2015/051130, filed on Sep. 21, 2015, and entitled“Dynamic Input Scaling for Controls of Robotic Surgical System,” andInternational Patent Application No. PCT/US2016/14031, filed Jan. 20,2016, the entire contents of each of these disclosures is hereinincorporated by reference.

Referring also to FIGS. 2 and 3 , the rotation of the input device 46relative to each of the X, Y, and Z coordinate axes may be scaled torotation of the tool 20 about a roll axis “R”, a pitch axis “P”, and ayaw axis “Y” (RPY). The roll axis “R” is aligned with an end effector ofa tool, e.g., tool 20, as displayed on the display device 44 while pitchand yaw axes “P”, “Y” are orientated to the camera frame as displayed onthe display device 44 such that motions of the handles 42 and/or inputdevice 46 are relative to a clinician's view of the display device 44.Specifically, the pitch axis “P” is about the X coordinate axis and theyaw axis “Y” is about the Y coordinate axis of a neutral frame of thehandle. The scaling of rotation of the input device 46 about each of theRPY axes may be scaled up, down, or neutral manner. By scaling rotationup, a clinician is able to reduce rotation of the input device 46 abouta particular one of the RPY axes to achieve a desired rotation of thetool 20 about the respective RPY axis. This up scaling may allow aclinician to have dexterity beyond a natural movement of the human body.For example, a clinician may roll a tool 20 beyond what is possible withthe movement of the clinician's wrist without releasing the input device46. In contrast, by scaling rotation down, a clinician is able to moreprecisely control rotation of the tool 20 about a particular one of theRPY axes of the tool 20 in response to rotation of the input device 46.

Rotation of the input device 46 about each of the RPY axes may be scaledin a different manner to rotation of the tool 20. For example, rotationof the input device 46 about the control shaft 43, i.e., rotation aboutthe roll axis “R”, may be scaled up, rotation of the input device 46about the pitch axis “P” may be scaled in a neutral manner, and rotationof the input device 46 about the yaw axis “Y” may be scaled down. Anyother combinations of scaling are contemplated herein and form a part ofthe present disclosure.

During a surgical procedure, rotation of the input device 46 may be“clutched” relative to rotation of the tool 20 about the RPY axes. The“clutching” of the input device 46 relative to the tool 20 may bemanually selected by a clinician or may be automatically selected by therobotic surgical system 1. When input device 46 is reassociated or“declutched” with rotation of the tool 20 about the RPY axes, the inputdevice 46 may be misaligned with the orientation of the tool 20 aboutone or more of the RPY axes.

The robotic surgical system 1 may vary the RPY scaling factors torealign the input device 46 with the tool 20 in a manner which isimperceptible to a clinician engaged with the input device 46. Torealign the input device 46 with the tool 20 the RPY scaling factors ina direction away from an aligned or centered position may be scaled downmore than when the clinician moves the input handle 46 towards thealigned configuration until the tool is aligned with the input device46. By scaling down movement of the input handle 46 as the input handle46 is moved towards the aligned configuration, the robotic surgicalsystem 10 allows the input device 46 to “catch up” to the position ofthe tool 20 in a manner which is indiscernible to a clinicianinterfacing with the input device 46. For example, one or more of theRPY scaling factors may be increased in a range of about 5% to about200% (e.g., 25% or 50%) when the input device 46 is moved away from thealigned configuration and the RPY scaling factor may remain unchangedwhen the input device 46 is moved towards the aligned configuration.Additionally or alternatively, one or more of the RPY scaling factorsmay be reduced in a range of about 5% to about 200% (e.g., 25% or 50%)when the input device 46 is moved towards the aligned configurationwhich causes the tool 20 to catch up with the position of the inputdevice 46. When the tool 20 is aligned with the input device 46, the RPYscaling factors return to operating in a symmetrical manner which may beup, down, or neutral, e.g., have the same scaling factor. It will beappreciated that by scaling movement of the input device 46 relative tothe tool 20 in this manner, the tool 20 remains stationary when theinput device 46 is stationary and tool 20 only moves in response to aclinician moving the input device 46.

In an embodiment, the RPY scaling factors may vary based on the amountof misalignment of the respective RPY axis. For example, when the rollaxis “R” is misaligned about 15 degrees, the scaling factor of the rollaxis “R” may be scaled down about 10% for movement of the input device46 towards the aligned configuration, and when the roll axis “R” ismisaligned about 30 degrees, the roll axis “R” may be scaled down about20% for movement of the input device 46 towards the alignedconfiguration. It is contemplated that the varying of the scalingfactors may be a linear, exponential, polynomial, linear step, or othermathematical relationship. For a detailed description of varying scalingfactors based on a distance or amount of misalignment, reference can bemade to International Patent Application No. PCT/US16/14031, filed Jan.20, 2016, the entire contents of which are hereby incorporated byreference.

In some embodiments, the robotic surgical system 1 may vary the RPYscaling factors to realign the input device 46 with the tool 20 untilthe misalignment of the input device 46 with the tool 20 is at or withina predetermined offset. For example, the input device 46 may bemisaligned with the tool 20 about a roll axis by 100°. The roboticsurgical system 1 may vary the roll scaling factors (e.g., increase theroll scaling factor when the input device 46 is moved away from analigned configuration and/or decrease the roll scaling factor when theinput device 46 is moved towards the aligned configuration) until theinput device 46 is at a predetermined offset or “trim” in a range ofabout 10° to about 45° (e.g., about 30°) relative to the input device46. Once the input device 46 is at the predetermined offset relative tothe input device 46, the robotic surgical system 1 ceases to vary theroll scaling factors to realign the input device 46 with the tool 20.

By allowing the input device 46 to remain offset from the tool 20 aboutone or more of the RPY axes, the position may allow the clinician tomaintain a more comfortable hand and/or arm position during the surgicalprocedure. The predetermined offset may be the same or different abouteach of the RPY axes. For example, the predetermined offset may be in arange of about 5° to about 45° (e.g., about 15°) about each of the RPYaxes. Alternatively, the predetermined offset about the roll axis may beabout 30° and the predetermined offset about the pitch and yaw axes maybe about 15°.

The predetermined offsets may be set by the robotic surgical system 1 ormay be user selected. The predetermined offsets may be set by therobotic surgical system 1 based on the type of tool 20 connected to anarm 12 associated with the input device 46 and/or may be set by therobotic surgical system 1 based on the type of surgical procedure. Itwill be appreciated that as the predetermined offset increases, thecontrol of the tool 20 may become more difficult. Accordingly, there maybe limits to the maximum limits set for the user selected predeterminedoffsets.

Referring to FIGS. 3 and 4 , it may be beneficial to “flip” one or moreof the RPY axes if the misalignment is greater than a predeterminedmisalignment. One or more of the RPY axes is “flipped” when the alignedconfiguration about a respective axis is rotated 180° about therespective axis. A “flip” about a respective axis may be beneficial forcontrol of a tool 20 when the tool 20 is asymmetrical or directional(e.g., a hook, curved scissors, curved dissector, etc.). For example,when a tool 20 is a hook with an opening 21 facing right as shown on thedisplay device 44 in FIG. 3 , the tool 20 can be “flipped” about theroll axis “R” such that the opening 21 faces left as shown on thedisplay device 44 as shown in FIG. 4 . The predetermined misalignmentmay be in a range of about 125° to about 195° (e.g., about 135°).Additionally or alternatively, a clinician may “flip” one or more of theRPY axes using a button, voice command, or GUI input. It is contemplatedthe robotic surgical system 1 may incorporate both a “trim” and a “flip”algorithm.

Additionally or alternatively, the robotic surgical system 1 may includea “snap” algorithm which is similar to the “flip” algorithm. A “snap”would occur when one or more of the RPY axes is misaligned greater thana predetermined misalignment. For example, when the roll axis “R” ismisaligned about 45°, the trim of the roll axis “R” may snap to anoffset of 45°. The snap algorithm may include a plurality of sequentialoffsets such that each time the particular RPY axes exceeds apredetermined misalignment, the respective RPY axis “snaps” to an offsetassociated with the predetermined misalignment. For example, the rollaxis “R” may have predetermined snap points spaced 15, 30, or 45 degreesabout the roll axis “R”.

Referring back to FIG. 2 , the input device 46 includes a button 47 toalter the scaling of one or more of the RPY scaling factors. Forexample, when the button 47 is depressed, the scaling factor about theroll axis “R” can be scaled up or scaled down to a predetermined value.Alternatively, when the button 47 is depressed, the input device 46 canbe clutched out about the roll axis “R” while the other axes remainrelated to movement of the input device 46.

While several embodiments of the disclosure have been shown in thedrawings, it is not intended that the disclosure be limited thereto, asit is intended that the disclosure be as broad in scope as the art willallow and that the specification be read likewise. Any combination ofthe above embodiments is also envisioned and is within the scope of theappended claims. Therefore, the above description should not beconstrued as limiting, but merely as exemplifications of particularembodiments. Those skilled in the art will envision other modificationswithin the scope of the claims appended hereto.

What is claimed:
 1. A method of operating a surgical robot or surgicalsimulator, the method comprising: receiving, with a processing unit, arotation of an input device of a robotic surgical system about a firstaxis of rotation; and scaling the rotation of the input device to arotation of a representation of a tool on a display device about a firstaxis of movement, the processing unit down scaling rotation of the inputdevice when the input device is moved away from an aligned configurationto realign the input device with the representation of a tool until theinput device is within a predetermined offset with the representation ofa tool about the first axis of rotation.
 2. The method according toclaim 1, wherein scaling rotation of the input device to the rotation ofthe representation of a tool on the display device about the first axisof movement includes the processing unit up scaling rotation of theinput device when the input device is moved towards the alignedconfiguration to realign the input device with the representation of atool until the input device is within the predetermined offset with therepresentation of a tool about the first axis of rotation.
 3. The methodaccording to claim 1, further comprising selecting the predeterminedoffset.
 4. The method according to claim 3, wherein selecting thepredetermined offset includes the processing unit determining thepredetermined offset based on the representation of a tool.
 5. Themethod according to claim 1, further comprising flipping the alignedconfiguration of the input device 180° about the first axis of rotationwhen the input device is misaligned with the representation of the toolgreater than a predetermined misalignment.